xxxxxxxxxx::header{start fixed="false"}::header{start fixed="false"}
<img src="
https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png"
width="300">
::header{end}
::page{title="Set Up a Simple Gradio Interface to Interact with Your Models"}
Estimated time needed: 30 minutes
Imagine you\'re developing a customer support chatbot for a company. The goal is to provide a seamless and interactive way for customers to get answers to their questions. To achieve this, you need an interface where users can input their queries and receive responses generated by an large language model (LLM). This lab will guide you through creating such an interface using Gradio. You\'ll learn how to integrate various Gradio components, like text input fields, buttons, and display elements, to create an intuitive and user-friendly experience.
Creating an interface is crucial for several reasons:
- User accessibility: A front-end interface makes it easy for users to interact with the LLM without needing technical expertise.
- Enhanced user experience: An intuitive interface can provide a better user experience, making interactions smoother and more efficient.
- Customization: Gradio allows you to customize the interface to meet specific needs, whether for a chatbot, a data analysis tool, or other applications.
- Seamless integration: Gradio\'s flexibility enables you to seamlessly integrate the front-end with various backend technologies, including LLMs.
<div style="text-align: center;">
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/f3OYckJ0KIZxIb9zsCoZ8A/gradio%20cover.png" alt="Alt text" width="60%" height="auto">
<p>Source: DALL-E</p>
</div>
In this lab, you will learn how to use Gradio to set up a front-end interface that allows users to interact with a backend LLM. Gradio is a powerful and user-friendly library for creating customizable web-based interfaces. By the end of this lab, you will have the skills to use essential and commonly used Gradio elements to build an interface that communicates with an LLM, enabling you to create a functional chatbot.
This lab will provide you with hands-on experience in building an interactive interface, empowering you to create applications that leverage the power of LLMs through user-friendly front-end solutions.
## Learning objectives
By the end of this project, you will be able to:
- **Use Gradio to build interactive front-end interfaces**, enabling users to interact with backend LLMs seamlessly
- **Create a functional chatbot**, allowing users to input queries and receive responses from an LLM
- **Implement essential and commonly used Gradio elements**, such as text input fields, buttons, and display areas, to enhance user experience
- **Customize and deploy web-based applications**, facilitating various use cases like customer support, data analysis, and more
::page{title="Setup"}
## Setting up a virtual environment
Let\'s create a virtual environment. Using a virtual environment allows you to manage dependencies for different projects separately, avoiding conflicts between package versions.
In the terminal of your Cloud IDE, ensure that you are in the path `/home/project`, then run the following commands to create a Python virtual environment.
```shell
pip install virtualenv
virtualenv my_env # create a virtual environment named my_env
source my_env/bin/activate # activate my_env
```
## Installing necessary libraries
To ensure a seamless execution of our scripts, and considering that certain functions within these scripts rely on external libraries, it\'s essential to install some prerequisite libraries before you begin. For this project, the key libraries you will need are `Gradio` for creating user-friendly web interfaces and `IBM Watsonx AI` for leveraging advanced LLM model from IBM watsonx\'s API.
- **[`gradio`](https://www.gradio.app/)** allows you to build interactive web applications quickly, making your AI models accessible to users with ease.
- **[`ibm-watsonx-ai`](https://ibm.github.io/watsonx-ai-python-sdk/)** for using LLMs from IBM's watsonx.ai.
- **[`langchain`, `langchain-ibm`, `langchain-community`](https://www.langchain.com/)** for using relevant features from LangChain.
Here\'s how to install these packages (still from your terminal):
```shell
# installing necessary pacakges in my_env
python3.11 -m pip install \
gradio==4.44.0 \
ibm-watsonx-ai==1.1.2 \
langchain==0.2.11 \
langchain-community==0.2.10 \
langchain-ibm==0.1.11
```
Now, the environment is ready to create Python files.
::page{title="Quick tutorial to Gradio"}
## Gradio overview
Gradio can be used to create a web interface to seamlessly bridge the gap between machine learning models and end-user interaction. It enables developers to effortlessly transform their algorithms into accessible, interactive applications. With Gradio, you can quickly design web interfaces that allow users to input data in various forms—such as text, images, or audio—and instantly view the output generated by your model. For more information, please visit [Gradio](https://www.gradio.app/).
In this project, your LLM-based resume advisor application will need a web interface to interact with. So, let\'s get familiar with Gradio and create a simple Gradio application first.
## Create a Gradio demo
### Create a .py file
First, you need to create a .py file in cloud IDE on your left. Under the `EXPLORER` tab, open the `PROJECT` directory, right click and choose `New file`.
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/kmm9uO3_t2mFPGr6JtnJXQ/createfile.png" alt="Alt text" width="50%" height="50%"></br>
Then, you can name your file `gradio_demo.py`.
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/71V6X2-L0NSA4yRt7MFUHA/newfile.png" alt="Alt text" width="50%" height="50%"></br>
### A demo of sum calculator
You will create an application that can calculate the sum of your input numbers.
In the `gradio_demo.py`, you can enter the following code.
The `gr.Number()` element from gradio is being used to create a numeric field for the user to enter numbers as input or display numeric output.
```python
import gradio as gr
def add_numbers(Num1, Num2):
return Num1 + Num2
# Define the interface
demo = gr.Interface(
fn=add_numbers,
inputs=[gr.Number(), gr.Number()], # Create two numerical input fields where users can enter numbers
outputs=gr.Number() # Create numerical output fields
)
# Launch the interface
demo.launch(server_name="127.0.0.1", server_port= 7860)
```
### Launch the application
Return to the terminal and verify that the virtual environment `my_env` label appears at the start of the line, which means that you are in the `my_env` environment that you just created. Next, execute the following command to run the Python script.
```shell
python3.11 gradio_demo.py
```
After it runs successfully, you will see a message similar to following in the terminal:
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/c8yfluESyjbIFHM8Q_Lwng/excute.png" alt="Alt text" width="50%" height="50%"></br>
Since the web application is hosted locally on port 7860, click on the following button to view the application we\'ve developed.
::startApplication{port="7860" display="internal" name="Web Application" route="/"}
(Note: if this "Web Application" button does not work, follow the following picture instructions to launch the app.")
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/nmKSkEpdjKlU7Sxg4awyog/app.png" alt="Alt text" width="25%" height="50%"></br>
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/9AK21jhDZ6iUf59nSEGHcA/app2.png" alt="Alt text" width="50%" height="50%"></br>
Let\'s take a look at the web application. It shows an example of sum of 3 and 4. You\'re encouraged to experiment with the web app\'s inputs and outputs!
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/PQYvaZM9SEsBX5i9TiywuQ/gradiodemo.png" alt="Alt text" width="50%" height="50%"></br>
If you want to stop the script, you can press `Ctrl+C` in the terminal and close the appliction window.
### Exercise
Can you create a Gradio application that can combine two input sentences together? Take your time to complete this exercise.
<details>
<summary>Let's take a look of the answer.</summary>
import gradio as gr
def combine(a, b):
return a + " " + b
demo = gr.Interface(
fn=combine,
inputs = [
gr.Textbox(label="Input 1"),
gr.Textbox(label="Input 2")
],
outputs = gr.Textbox(value="", label="Output")
)
demo.launch(server_name="127.0.0.1", server_port= 7860)
</details>
You just had a first taste of the Gradio interface; it\'s easy right? If you want to learn more about customization in Gradio, you can take this guided project **[Bring your Machine Learning model to life with Gradio](https://cognitiveclass.ai/courses/bring-your-machine-learning-model-to-life-with-gradio)**. You can find more relevant courses and projects on [cognitiveclass.ai](https://cognitiveclass.ai/)
For the rest of this project, you will use Gradio as an interface for the created application.
::page{title="Quick start on watsonx.ai LLM"}
## watsonx.ai introduction
watsonx.ai is IBM\'s commercial generative AI and scientific data platform based on cloud technology. It encompasses a studio, data store, and governance toolkit, designed to support multiple LLMs. This platform is tailored for a wide range of AI development tasks, offering developers access to IBM\'s own series of LLMs as well as models from Meta\'s Llama-3 and Mixtral model.
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/5bjV6F3Dzn5tLzp3UQWn0w/watsonx.png" alt="Alt text" width="55%" height="50%"></br>
In this section, you will be guided through the process of using watsonx.ai\'s API to create a simple Q&A bot. This bot will leverage the advanced capabilities of watsonx.ai to understand and respond to user queries accurately. Whether you\'re new to programming or an experienced developer, this step-by-step tutorial will equip you with the knowledge to integrate watsonx.ai\'s LLMs into your applications.
## Create a Q&A bot
This tutorial will walk you through the process of creating a Q&A chatbot leveraging the `mistralai/mixtral-8x7b-instruct-v01` model developed by Mistral AI. This powerful foundation model has been seamlessly integrated into IBM\'s watsonx.ai platform, simplifying your development journey. The provided API eliminates the need for generating complex API tokens, streamlining the application creation process. The `mistralai/mixtral-8x7b-instruct-v01` model is equipped with the following features:
- Supports Q&A
- Summarization
- Classification
- Generation
- Extraction
- Retrieval-augmented generation
- Code generation
This model aligns perfectly with your objective for the application being created. For models and their IDs, please refer to [here](https://ibm.github.io/watsonx-ai-python-sdk/fm_model.html).
**Note**: The `mistralai/mixtral-8x7b-instruct-v01` model, like any AI technology, has its limitations, and it is possible to encounter nonsensical responses occasionally. The primary objective of this project is to provide guidence on utilizing LLMs with watsonx.ai.
Follow these step-by-step instructions to create your application:
1. Still in the `PROJECT` directory, create a new Python file named `simple_llm.py` (you are welcome to choose a different name if you prefer).
2. Enter the following script content into your newly created `simple_llm.py` file and save your changes. Line-by-line explanation of the code snippet is provided.
```python
# Import necessary packages
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai import Credentials
from langchain_ibm import WatsonxLLM
# Model and project settings
model_id = 'mistralai/mixtral-8x7b-instruct-v01' # Directly specifying the model
# Set necessary parameters
parameters = {
GenParams.MAX_NEW_TOKENS: 256, # Specifying the max tokens you want to generate
GenParams.TEMPERATURE: 0.5, # This randomness or creativity of the model's responses
}
project_id = "skills-network"
# Wrap up the model into WatsonxLLM inference
watsonx_llm = WatsonxLLM(
model_id=model_id,
url="https://us-south.ml.cloud.ibm.com",
project_id=project_id,
params=parameters,
)
# Get the query from the user input
query = input("Please enter your query: ")
# Print the generated response
print(watsonx_llm.invoke(query))
```
3. Open your terminal and ensure that you are operating within the virtual environment (`my_env`) you previously established.
4. Run the following command in the terminal.
```bash
python3.11 simple_llm.py
```
Upon successful execution of the code, you can input our query in the terminal. A response is then generated.
The following picture is showing an example response based on the query: `How to be a good data scientist?`.
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/a2Tj3VWxnAccg4KqIshiyg/be%20ds.png" alt="Alt text" width="50%" height="50%"></br>
In the code, you simply used `"skills-network"` as `project_id` to gain immediate, free access to the API without the need for initial registration. It\'s important to note that this access method is exclusive to this cloud IDE environment. If you are interested in using the model/API in a local environment, detailed instructions and further information are available in this [tutorial](https://medium.com/the-power-of-ai/ibm-watsonx-ai-the-interface-and-api-e8e1c7227358).
::page{title="Gradio and LLM"}
## Integrate the application into Gradio
Having successfully created a Q&A bot with your script, you might notice that responses are only displayed in the terminal. You may wonder if it\'s possible to integrate this application with Gradio to leverage a web interface for inputting questions and receiving responses.
The following code guides you through this integration process. It includes three components:
- Initializing the model
- Defining the function that generates responses from the LLM
- Constructing the Gradio interface, enabling interaction with the LLM
- The `gr.Textbox` element is being used to create text field and hold users\' input query and LLM's output.
```python
# Import necessary packages
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai import Credentials
from langchain_ibm import WatsonxLLM
import gradio as gr
# Model and project settings
model_id = 'mistralai/mixtral-8x7b-instruct-v01' # Directly specifying the model
# Set necessary parameters
parameters = {
GenParams.MAX_NEW_TOKENS: 256, # Specifying the max tokens you want to generate
GenParams.TEMPERATURE: 0.5, # This randomness or creativity of the model's responses
}
project_id = "skills-network"
# Wrap up the model into WatsonxLLM inference
watsonx_llm = WatsonxLLM(
model_id=model_id,
url="https://us-south.ml.cloud.ibm.com",
project_id=project_id,
params=parameters,
)
# Function to generate a response from the model
def generate_response(prompt_txt):
generated_response = watsonx_llm.invoke(prompt_txt)
return generated_response
# Create Gradio interface
chat_application = gr.Interface(
fn=generate_response,
allow_flagging="never",
inputs=gr.Textbox(label="Input", lines=2, placeholder="Type your question here..."),
outputs=gr.Textbox(label="Output"),
title="Watsonx.ai Chatbot",
description="Ask any question and the chatbot will try to answer."
)
# Launch the app
chat_application.launch(server_name="127.0.0.1", server_port= 7860)
```
1. Navigate to the `PROJECT` directory, right-click, and create a new file named `llm_chat.py`.
2. Input the script provided above into this new file.
3. Open your terminal and ensure you are within the `my_env` virtual environment.
4. Execute the following code in the terminal to run the application.
```bash
python3.11 llm_chat.py
```
After it has excuted successfully, you will see message similar to the following in the terminal:
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/c8yfluESyjbIFHM8Q_Lwng/excute.png" alt="Alt text" width="50%" height="50%"></br>
5. Click the following button to launch and view the application.
::startApplication{port="7860" display="internal" name="Web Application" route="/"}
The chatbot you have successfully created will be displayed, appearing as follows:
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/nbSA43ABX3btuUL9shx4Yw/chatbot-interface-.png" alt="Alt text" width="80%" height="50%"></br>
Now, feel free to ask any question to the chatbot.
Here is an example of the question asked:
<img src="https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/l2l5J3OfYqlM_dG1wrOjUA/chatbot-response-.png" alt="Alt text" width="80%" height="50%"></br></br>
(To terminate the script, press `Ctrl+C` in the terminal and close the appliction window.)
### Exercise
You might observe that the responses from the LLM are occasionally incomplete. Could you identify the cause of this issue? Also, would you be able to modify the code to enable the model to generate more content?
Actually, all you need to do is:
<details>
<summary>Click here for the answer</summary>
"max_new_tokens": 512, # adjust the parameter `max_new_token` to a bigger value
</details>
In the following section, we will explore how to develop a resume polisher using the knowledge we have just acquired.
::page{title="Authors"}
## Authors
[Kang Wang](https://author.skills.network/instructors/kang_wang)
Kang Wang is a data scientist in IBM. He is also a PhD candidate in the University of Waterloo.
## Other contributors
[Joseph Santarcangelo](https://author.skills.network/instructors/joseph_santarcangelo)
Joseph has a Ph.D. in electrical engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.
<!--## Change log
|Date (YYYY-MM-DD)|Version|Changed By|Change Description|
|-|-|-|-|
|2024-07-30|0.1|Kang Wang|Create the lab|
|-|-|-|-|
|2024-08-09|-|PRashant Juyal|QA|
-->
Copyright © IBM Corporation. All rights reserved.

Estimated time needed: 30 minutes
Imagine you're developing a customer support chatbot for a company. The goal is to provide a seamless and interactive way for customers to get answers to their questions. To achieve this, you need an interface where users can input their queries and receive responses generated by an large language model (LLM). This lab will guide you through creating such an interface using Gradio. You'll learn how to integrate various Gradio components, like text input fields, buttons, and display elements, to create an intuitive and user-friendly experience.
Creating an interface is crucial for several reasons:
Source: DALL-E
In this lab, you will learn how to use Gradio to set up a front-end interface that allows users to interact with a backend LLM. Gradio is a powerful and user-friendly library for creating customizable web-based interfaces. By the end of this lab, you will have the skills to use essential and commonly used Gradio elements to build an interface that communicates with an LLM, enabling you to create a functional chatbot.
This lab will provide you with hands-on experience in building an interactive interface, empowering you to create applications that leverage the power of LLMs through user-friendly front-end solutions.
By the end of this project, you will be able to:
Let's create a virtual environment. Using a virtual environment allows you to manage dependencies for different projects separately, avoiding conflicts between package versions.
In the terminal of your Cloud IDE, ensure that you are in the path /home/project, then run the following commands to create a Python virtual environment.
- 1
- 2
- 3
pip install virtualenvvirtualenv my_env # create a virtual environment named my_envsource my_env/bin/activate # activate my_env
To ensure a seamless execution of our scripts, and considering that certain functions within these scripts rely on external libraries, it's essential to install some prerequisite libraries before you begin. For this project, the key libraries you will need are Gradio for creating user-friendly web interfaces and IBM Watsonx AI for leveraging advanced LLM model from IBM watsonx's API.
gradio allows you to build interactive web applications quickly, making your AI models accessible to users with ease.
ibm-watsonx-ai for using LLMs from IBM’s watsonx.ai.
langchain, langchain-ibm, langchain-community for using relevant features from LangChain.
Here's how to install these packages (still from your terminal):
- 1
- 2
- 3
- 4
- 5
- 6
- 7
# installing necessary pacakges in my_envpython3.11 -m pip install \gradio==4.44.0 \ibm-watsonx-ai==1.1.2 \langchain==0.2.11 \langchain-community==0.2.10 \langchain-ibm==0.1.11
Now, the environment is ready to create Python files.
Gradio can be used to create a web interface to seamlessly bridge the gap between machine learning models and end-user interaction. It enables developers to effortlessly transform their algorithms into accessible, interactive applications. With Gradio, you can quickly design web interfaces that allow users to input data in various forms—such as text, images, or audio—and instantly view the output generated by your model. For more information, please visit Gradio.
In this project, your LLM-based resume advisor application will need a web interface to interact with. So, let's get familiar with Gradio and create a simple Gradio application first.
First, you need to create a .py file in cloud IDE on your left. Under the EXPLORER tab, open the PROJECT directory, right click and choose New file.
Then, you can name your file gradio_demo.py.
You will create an application that can calculate the sum of your input numbers.
In the gradio_demo.py, you can enter the following code.
The gr.Number() element from gradio is being used to create a numeric field for the user to enter numbers as input or display numeric output.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
import gradio as grdef add_numbers(Num1, Num2):return Num1 + Num2# Define the interfacedemo = gr.Interface(fn=add_numbers,inputs=[gr.Number(), gr.Number()], # Create two numerical input fields where users can enter numbersoutputs=gr.Number() # Create numerical output fields)# Launch the interfacedemo.launch(server_name="127.0.0.1", server_port= 7860)
Return to the terminal and verify that the virtual environment my_env label appears at the start of the line, which means that you are in the my_env environment that you just created. Next, execute the following command to run the Python script.
- 1
python3.11 gradio_demo.py
After it runs successfully, you will see a message similar to following in the terminal:
Since the web application is hosted locally on port 7860, click on the following button to view the application we've developed.
(Note: if this “Web Application” button does not work, follow the following picture instructions to launch the app.”)
Let's take a look at the web application. It shows an example of sum of 3 and 4. You're encouraged to experiment with the web app's inputs and outputs!
If you want to stop the script, you can press Ctrl+C in the terminal and close the appliction window.
Can you create a Gradio application that can combine two input sentences together? Take your time to complete this exercise.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
import gradio as grdef combine(a, b):return a + " " + bdemo = gr.Interface(fn=combine,inputs = [gr.Textbox(label="Input 1"),gr.Textbox(label="Input 2")],outputs = gr.Textbox(value="", label="Output"))demo.launch(server_name="127.0.0.1", server_port= 7860)
You just had a first taste of the Gradio interface; it's easy right? If you want to learn more about customization in Gradio, you can take this guided project Bring your Machine Learning model to life with Gradio. You can find more relevant courses and projects on cognitiveclass.ai
For the rest of this project, you will use Gradio as an interface for the created application.
watsonx.ai is IBM's commercial generative AI and scientific data platform based on cloud technology. It encompasses a studio, data store, and governance toolkit, designed to support multiple LLMs. This platform is tailored for a wide range of AI development tasks, offering developers access to IBM's own series of LLMs as well as models from Meta's Llama-3 and Mixtral model.
In this section, you will be guided through the process of using watsonx.ai's API to create a simple Q&A bot. This bot will leverage the advanced capabilities of watsonx.ai to understand and respond to user queries accurately. Whether you're new to programming or an experienced developer, this step-by-step tutorial will equip you with the knowledge to integrate watsonx.ai's LLMs into your applications.
This tutorial will walk you through the process of creating a Q&A chatbot leveraging the mistralai/mixtral-8x7b-instruct-v01 model developed by Mistral AI. This powerful foundation model has been seamlessly integrated into IBM's watsonx.ai platform, simplifying your development journey. The provided API eliminates the need for generating complex API tokens, streamlining the application creation process. The mistralai/mixtral-8x7b-instruct-v01 model is equipped with the following features:
This model aligns perfectly with your objective for the application being created. For models and their IDs, please refer to here.
Note: The mistralai/mixtral-8x7b-instruct-v01 model, like any AI technology, has its limitations, and it is possible to encounter nonsensical responses occasionally. The primary objective of this project is to provide guidence on utilizing LLMs with watsonx.ai.
Follow these step-by-step instructions to create your application:
Still in the PROJECT directory, create a new Python file named simple_llm.py (you are welcome to choose a different name if you prefer).
Enter the following script content into your newly created simple_llm.py file and save your changes. Line-by-line explanation of the code snippet is provided.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
# Import necessary packagesfrom ibm_watsonx_ai.foundation_models import ModelInferencefrom ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParamsfrom ibm_watsonx_ai import Credentialsfrom langchain_ibm import WatsonxLLM# Model and project settingsmodel_id = 'mistralai/mixtral-8x7b-instruct-v01' # Directly specifying the model# Set necessary parametersparameters = {GenParams.MAX_NEW_TOKENS: 256, # Specifying the max tokens you want to generateGenParams.TEMPERATURE: 0.5, # This randomness or creativity of the model's responses}project_id = "skills-network"# Wrap up the model into WatsonxLLM inferencewatsonx_llm = WatsonxLLM(model_id=model_id,url="https://us-south.ml.cloud.ibm.com",project_id=project_id,params=parameters,)# Get the query from the user inputquery = input("Please enter your query: ")# Print the generated responseprint(watsonx_llm.invoke(query))
Open your terminal and ensure that you are operating within the virtual environment (my_env) you previously established.
Run the following command in the terminal.
- 1
python3.11 simple_llm.py
Upon successful execution of the code, you can input our query in the terminal. A response is then generated.
The following picture is showing an example response based on the query: How to be a good data scientist?.
In the code, you simply used "skills-network" as project_id to gain immediate, free access to the API without the need for initial registration. It's important to note that this access method is exclusive to this cloud IDE environment. If you are interested in using the model/API in a local environment, detailed instructions and further information are available in this tutorial.
Having successfully created a Q&A bot with your script, you might notice that responses are only displayed in the terminal. You may wonder if it's possible to integrate this application with Gradio to leverage a web interface for inputting questions and receiving responses.
The following code guides you through this integration process. It includes three components:
gr.Textbox element is being used to create text field and hold users' input query and LLM’s output.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
# Import necessary packagesfrom ibm_watsonx_ai.foundation_models import ModelInferencefrom ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParamsfrom ibm_watsonx_ai import Credentialsfrom langchain_ibm import WatsonxLLMimport gradio as gr# Model and project settingsmodel_id = 'mistralai/mixtral-8x7b-instruct-v01' # Directly specifying the model# Set necessary parametersparameters = {GenParams.MAX_NEW_TOKENS: 256, # Specifying the max tokens you want to generateGenParams.TEMPERATURE: 0.5, # This randomness or creativity of the model's responses}project_id = "skills-network"# Wrap up the model into WatsonxLLM inferencewatsonx_llm = WatsonxLLM(model_id=model_id,url="https://us-south.ml.cloud.ibm.com",project_id=project_id,params=parameters,)# Function to generate a response from the modeldef generate_response(prompt_txt):generated_response = watsonx_llm.invoke(prompt_txt)return generated_response# Create Gradio interfacechat_application = gr.Interface(fn=generate_response,allow_flagging="never",inputs=gr.Textbox(label="Input", lines=2, placeholder="Type your question here..."),outputs=gr.Textbox(label="Output"),title="Watsonx.ai Chatbot",description="Ask any question and the chatbot will try to answer.")# Launch the appchat_application.launch(server_name="127.0.0.1", server_port= 7860)
PROJECT directory, right-click, and create a new file named llm_chat.py.
my_env virtual environment.
- 1
python3.11 llm_chat.py
After it has excuted successfully, you will see message similar to the following in the terminal:
The chatbot you have successfully created will be displayed, appearing as follows:
Now, feel free to ask any question to the chatbot.
Here is an example of the question asked:
(To terminate the script, press Ctrl+C in the terminal and close the appliction window.)
You might observe that the responses from the LLM are occasionally incomplete. Could you identify the cause of this issue? Also, would you be able to modify the code to enable the model to generate more content?
Actually, all you need to do is:
- 1
"max_new_tokens": 512, # adjust the parameter `max_new_token` to a bigger value
In the following section, we will explore how to develop a resume polisher using the knowledge we have just acquired.
Kang Wang is a data scientist in IBM. He is also a PhD candidate in the University of Waterloo.
Joseph has a Ph.D. in electrical engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.
Copyright © IBM Corporation. All rights reserved.